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ABSTRACT 


This  research  memorandum  determines 
the  relations  between  sustaining  engineer¬ 
ing  (SE)  usage,  methods  of  management,  and 
system  quality.  Principal  components  anal¬ 
ysis  was  used  to  combine  multiple  measures 
of  effectiveness  into  a  single  measure  of 
aircraft  quality  and  multiple  program 
characteristics  into  a  single  measure  of 
high-level  management.  Levels  of  SE  usage 
were  used  with  these  measures  to  determine 
the  relations,  if  any. 


-l- 


Accesslon  For 

NTIS  GRAAI 
DTIC  TAB 
Unannounced 
Just  If  lo*t  Ion. 


By - 

Distribution/ 


Availability  Codas 


Diat 


Avail  and/or 
Special 


□  □ 


TABLE  OF  CONTENTS 


Page 


List  of  Illustrations  .  v 

List  of  Tables  . vii 

Summary  .  1 

Introduction  .  1 

Analysis 


Measuring  Sustaining  Engineering  Usage  . 

Measuring  "System  Quality"  . 

Variables  Used  to  Measure  "Management"  . 

Results  of  Principal  Components  Analysis  for  "Management" 
Relations  Between  Sustaining  Engineering  Usage,  Quality, 


and  Management  .  12 

Conclusions  . 21 

Appendix  A:  Components  of  Sustaining  Engineering  .  A-1 

Appendix  B:  Annotated  Bibliography  of  Research  Memorandums 

in  the  Series  on  Sustaining  Engineering  .  B-1  -  B-5 

Appendix  C:  Principal  Components  Analyses  .  C-1  -  C-4 

Appendix  D:  Comparison  of  Principal  Components  Analyses 

Using  Binary  and  Scaled  Data  .  D-1  -  D-10 

Appendix  E:  Principal  Components  Analyses  for  Quality 

in  Different  Time  Periods  .  E-1  -  E-4 

Appendix  F:  Excursions  on  the  Statistical  Analysis  of  the 
Relation  Between  SE  Investments  Over  Time  and  Changes  in 
Quality  for  Related  Time  Periods  .  F-1  -  F-6 


-iii- 


cnM4-vo  O' 


LIST  OF  ILLUSTRATIONS 


Page 

1  Sampling  Distribution  of  F?  Values  Compared  to  Value 


Calculated  for  First  Principal  Component  .  11 

2  Aircraft  "Quality"  Compared  With  "Management"  .  13 

3  Aircraft  SE  Usage  Compared  With  "Management"  .  13 

4  Aircraft  SE  Usage  Compared  With  "Quality"  .  14 

5  Change  in  Quality:  fear  Two  vs.  Year  Six  .  16 

6  SE  Investments  From  Year  Two  Through  Year  Five  Compared 

With  Change  in  Quality  Between  Year  Two  and  Year  Six  .  20 


-v- 


LIST  OF  TABLES 


Page 

1  Relative  SE  Usage  .  4 

2  Results  of  the  Principal  Components  Analysis  for  "Quality" 

in  IQ-Type  Units  .  6 

3  Program  Characteristics  .  8 

4  Count  of  the  Number  of  "Yes"  Answers  per  Aircraft  in  the  Binary 

Program  Characteristics  Table  .  9 

s 

5  Results  of  Principal  Components  Analysis  for  "Management," 

in  IQ-Type  Units  .  11 

6  Comparing  "Quality"  and  "Management"  in  IQ- Type  Units 

and  SE  Usage  .  12 

7  Possible  Measures  of  SE  Usage  .  15 

8  A  Regression  Between  a  Change  in  Quality  and  SE  Investments  ....  17 

9  Results  of  Regression  Analysis  of  Changes  in  Quality  .  19 


-vii- 


SUMMARY 


•  '  One  of  the  major  elements  in  the  Vice  Chief  of  Naval  Operations 
(VCNO)  tasking  for  the  sustaining  engineering  (SE)  study  was  to  examine 
the  relations,  if  any,  between  SE  usage,  methods  of  management,  and  mea¬ 
sures  of  effectiveness  related  to  reliability,  maintainability,  and 
safety.  Another  research  memorandum  (RM)  in  this  series  estimates  rela¬ 
tive  SE  usage  among  a  large  sample  of  Navy  and  Air  Force  aircraft. 

Still  another  early  RM  in  this  series  examines  measures  of  effectiveness 
for  aircraft  reliability,  maintainability,  and  safety.  That  RM 
described  how  it  was  found  possible,  using  principal  components 
analysis,  to  combine  multiple  measures  of  effectiveness  into  a  single 
measure  of  aircraft  "quality." 

In  order  to  complete  the  comparisons  requested  in  the  tasking,  a 
measure  of  management  is  necessary.  The  same  technique  of  principal 
components  analysis  has  been  employed  to  combine  multiple  program  char¬ 
acteristics  into  a  single  measure  of  "management."  (These  program 
characterist ics  describe  the  type  of  "management  environment"  within 
which  a  program  office  works  and  are  not  an  indication  of  how  well  the 
program  office  itself  is  working.)  The  analysis  found  that  the  first 
principal  component  accounted  for  a  statistically  significant  percentage 
of  the  variation  in  the  program  characteristics,  thus  supporting  the 
idea  of  an  underlying  variable,  "management." 

The  three  single  measures  of  relative  SE  usage,  "quality,"  and 
"management,"  were  then  used  to  seek  out  relations,  if  any.  This  analy¬ 
sis  revealed  that  "quality"  and  "management"  show  a  significant  positive 
correlation.  "Management"  (as  defined  by  program  characteristics)  and 
overall  relative  SE  usage  show  no  obvious  correlation.  This  was  not 
surprising,  since  there  is  no  necessary  correlation  between  the  manage¬ 
ment  environment  created  by  high-level  choices  and  management  of  SE  at 
the  program  office  level. 

"Quality"  and  overall  relative  SE  usage  show  no  apparent  correla¬ 
tion.  SE  investments  in  a  particular  time  period  and  changes  in  "qual¬ 
ity"  related  to  that  time  period  do,  however,  show  a  positive 
correlation--which  is  consistent  with  the  general  belief  that  SE  can 
improve  quality. 

INTRODUCTION 

In  response  to  perceived  weaknesses  in  the  management  of  sustaining 
engineering  (SE)  services,  the  Vice  Chief  of  Naval  Operations  (VCNO) 
formally  tasked  the  Center  for  Naval  Analyses  (CNA)  with  a  study  of  the 
subject  in  June  1985.  Specifically,  the  VCNO  charged  CNA  with: 

•  Defining  SE  and  differentiating  it  from  other  forms  of 
weapon  system  engineering 
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•  Determining  how  much  the  Navy  spends  on  SE  and  what  it 
receives  for  those  expenditures 

•  Determining  how  much  SE  is  really  needed 

•  Determining  how  the  Navy  might  improve  its  control  of  SE . 

As  a  result,  CNA  developed  definitions  of  the  phases  of  weapon  sys¬ 
tem  engineering,  including  SE,  that  generally  have  been  accepted  within 
the  defense  industry.  The  three  basic  phases  are  defined  as  follows. 


•  Original  Design  Engineering:  All  engineering  effort, 
including  that  involved  in  development,  test,  and  evalua¬ 
tion,  leading  up  to  a  final  approved  weapon  system  design; 
that  is,  all  engineering  effort  occurring  before  comple¬ 
tion  of  the  Physical  Configuration  Audit,  when  design  is 
frozen  for  the  full-scale  production  article. 

•  Sustaining  Engineering:  Engineering  effort  involved  in 
setting  up  and  improving  production  processes,  ensuring 
systems  integration,  advising  on  and  correcting  deficien¬ 
cies  discovered  during  manufacturing  and  service  use,  and 
assisting  in  other  phases  of  manufacturing ,  such  as  qual¬ 
ity  assurance  and  configuration  control;  the  engineering 
required  to  ensure  that  curre  ,t  production  articles  meet 
explicit  or  implicit  specifications. 


•  Systems  Engineer ing :  Engineering  effort  applied  to  exist¬ 
ing  weapon  systems  intended  to  enhance  performance,  relia¬ 
bility,  or  maintainability  significantly  beyond  current 
specifications;  or  engineering  involved  in  the  design  of 
later,  improved  models  of  a  current  weapon  system. 

On  April  29,  1986,  Admiral  Busey  (then  the  VC NO)  was  briefed  on  the 
preliminary  results  of  the  CNA  study.  In  the  aggregate,  it  appeared 
that  approximately  17  million  man-hours  are  devoted  annually  to  SE  for 
aircraft  and  missiles  at  a  cost  to  the  Navy  of  approximately  $1  billion. 
It  was  found,  however,  that  accounting  and  budgeting  procedures  for  SE 
vary  widely  in  the  aerospace  industry  and  that  the  information  available 
on  the  specific  services  and  products  (i.e.,  outputs)  provided  by  SE  was 
not,  in  general,  sufficiently  detailed,  complete,  or  consistent  for  man¬ 
agerial  or  analytic  purposes.  Because  of  these  shortcomings,  the  study 
team  decided  to  focus  on  system  quality  rather  than  output  as  an  indica¬ 
tor  of  the  results  of  SE  efforts.  Because  the  shift  came  late  in  this 


1.  A  more  detailed  list  of  the  16  specific  tasks  constituting  SE  is 
provided  in  appendix  A. 
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first  phase  of  the  study,  the  work  was  incomplete  at  the  time  of  the 
April  briefing.  But  what  information  was  available  suggested  that 
relatively  moderate  levels  of  Sl.  hours  invested  per  aircraft  produced 
did  not  have  an  adverse  impact  on  aircraft  quality. 

These  and  other  findings  led  Admiral  Busey  to  give  CMA  a  revised 
and  expanded  tasking  in  May  1986.  One  of  the  more  important  elements  in 
this  tasking  was  to  analyze  the  relationship  between  SE  usage,  methods 
of  management,  and  measures  of  effectiveness  related  to  reliability, 
maintainability,  and  safety  (i.e.,  "system  quality"). 

To  answer  the  question  of  whether  SE  usage,  management,  and  system 
quality  are  related,  the  issues  of  measuring  SE  usage,  system  quality, 
and  management  had  to  be  addressed. 

ANALYSIS 

Measuring  Sustaining  Engineering  Usage 

Data  on  actual  sustaining  engineering  (SE)  hours  used  (or  estimated 
to  be  used  at  completion)  were  available,  by  annual  buys,  for  nine  dif¬ 
ferent  aircraft  during  the  period  1970  to  1986.  SE  hours  were  adjusted 
for  the  influence  of  a  number  of  determinants:  annual  production  quan¬ 
tities,  total  cumulative  production  (reflecting  a  kind  of  "learning 
curve"  effect),  model  changes  (which  interrupt  the  "learning  curve"), 
size  (with  airframe  weight  serving  as  a  proxy),  aircraft  manufacturing 
complexity  (with  physical  density  serving  as  a  proxy),  aircraft  design 
complexity  (with  maximum  mach  number  serving  as  a  proxy),  and  speed  of 
adjustment  to  changes  in  annual  production  quantities. 

Based  on  these  determinants,  regression  analysis  was  used  to  esti¬ 
mate  a  "target"  level  of  SE  hours  toward  which  actual  SE  hours  appeared 
to  oe  adjusting.  To  create  a  relative  measure  of  SE  usage  for  a  given 
aircraft,  total  actual  SE  hours  over  the  aircraft's  career  were  divided 
by  total  "target"  SE  hours  over  the  aircraft's  career.  (See  table  1.) 

A  value  below  1  means  that  SE  hours  expended  for  this  aircraft 
were,  on  average,  below  its  estimated  target;  a  value  above  1  means  that 
SE  hours  expended  were,  on  average,  above  the  estimated  target. 


1.  The  adjustment  of  SE  for  all  these  factors  was  made  using  regression 
analysis,  which  estimates  a  conditional  average  level  of  SE  (conditional 
on  quantities,  cumulative  production,  etc.)  based  on  industry-wide 
practi ce . 
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TABLE  1 


RELATIVE  SE  USAGE3 

Actual  SE  hours 
divided  by 

Aircraft  target  SE  hours 


F-5E 

0.83 

F—  1 4  A 

1 .04 

F-15A/B 

+  C/D 

1  .04 

F-16A/B 

+  C/D 

1 .06 

F-18A/B 

♦  C/D 

1.10 

A-6E 

0.82 

P-3C 

1 .06 

E-2C 

1 .23 

E-3 

1  .00 

a.  A  complete  description  of  this  work  and  the  underlying  data  can  be 
found  in  a  forthcoming  CRM  in  this  series  See  appendix  B. 


Measuring  "System  Quality"1 2 

As  a  start  in  measuring  quality,  data  were  compiled  on  a  wide  vari¬ 
ety  of  individual  indicators  of  aircraft  reliability,  maintainability, 
and  safety.  These  data  included  measures  of  maintenance  effort,  failure 
rates,  and  mishap  rates.  Each  of  these  variables  can  be  regarded  as  a 
directly  measurable,  but  incomplete,  indicator.  Consequently,  each  mea¬ 
sure  offers  only  a  partial  insight  into  some  overall  variable,  which  can 
be  characterized  as  "system  quality.  Quality,  it  is  surmised,  is  the 
latent,  unobservable  variable  that  finds  expression  in  the  values 


1.  For  a  more  complete  description  of  this  work  see  CRM  87-89,  Measures 
of  Aircraft  Reliability,  Maintainabi 1 i ty ,  and  Safety:  Do  They  Measure 
"System  Quality"?  Jun  1987. 

2.  It  should  be  noted  that  because  the  study  focused  on  the  use  of  SE  by 

prime  contractors,  data  for  all  the  measures  of  effectiveness  (except 
acceptance  testing  histories)  related  to  only  those  parts  of  the  air¬ 
craft,  or  its  mode  of  operation,  for  which  the  prime  contractor  and  his 
SE  input  are  responsible  "’hus,  maintenance  effort  or  failures  due  to 
GFE  engines  or  major  av:  cs  systems  would  not  be  included,  nor  would 

mishaps  due  to  such  fa'  -  as  pilot  error,  weather,  or  engine  malfunc¬ 
tions.  Essentially,  the  search  for  a  way  to  measure  quality  was  re- 
str;cted  to  the  aircraft  '  flying  machine,  sans  engines,  and  not  as  a 
complete  weapon  system 
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observed  for  each  of  the  specific  variables  noted  above. ^  To  measure 
quality  requires  a  "distillation"  of  the  observables  to  try  to  get  at 
the  underlying  variable  or  Vctrlaoies  that  generated  them. 

Principal  components  analysis  was  the  multivariate  statistical 
t-.chnique  selected  for  this  distillation  process.  Conceptually,  the 
technique  is  quite  straightforward.  A  weighted  average  score  is  com¬ 
puted  for  each  unit  of  observation  (each  type  of  aircraft  in  this  case), 
using  the  aircraft's  performance  on  each  of  several  partial  measures  of 
quality  (mean  flight  hours  between  failure  (MFHBF),  for  example).  The 
weights  given  to  each  measure  (or  variable)  are  not  arbitrarily  assigned 
or  chosen  a  priori  by  the  analyst,  but  are  determined  by  a  well-accepted 
statistical  technique  that  does  so  in  a  way  that  minimizes  the  unex¬ 
plained  variance  in  the  entire  data  set.  The  first  principal  component 
captures  the  largest  possible  percentage  of  the  total  variance.  The 
remaining  principal  components  are  chosen  to  best  account  for  what  vari¬ 
ance  remains  unexplained  by  the  first  principal  component. 

In  the  case  of  measures  of  aircraft  reliability,  maintainability, 
and  safety,  the  first  principal  component  accounted  for  85  percent  of 
the  variance.  The  statistical  significance  of  an  R1  2  of  85  percent  was 
checked,  using  a  numerically  generated  sampling  distribution  of  fl2 
values,  under  the  null  hypothesis  that  the  underlying  variables  are 
uncorrelated.  The  observed  R2  of  85  percent  was  highly  significant 
because  it  lies  7-1/2  standard  deviations  above  the  mean  of  the  null- 
hypothesis  sampling  distribution. 

Because  the  first  principal  component  scores  (interpreted  as  "qual¬ 
ity")  of  each  of  the  six  fighter/attack  aircraft  used  in  the  base  case 
analysis  have  no  natural  units  (they  are  measuring  an  unobservable), 
units  can  be  chosen  for  ease  of  interpretation.  Table  2  presents  the 


1.  The  term  "quality"  as  used  here  is  limited  to  airframe  reliability, 
maintainability,  and  safety--character istics  believed  to  be  affected  by 
SE  investments.  Quality  in  this  case  does  not  relate  to  the  aircraft's 
capabilities  for  performing  its  intended  missions;  nor  does  it  refer  to 
the  absence  or  presence  of  manufacturing  defects  and  sloppy  workmanship. 
Detecting  and  preventing  the  latter  is  primarily  the  work  of  quality 
assurance  teams  (which,  as  noted  earlier,  sometimes  do  require  SE 
inputs ) . 

2.  Attempting  to  distill  the  unobservable  overall  characteristic  "intel¬ 
ligence"  from  a  host  of  measurable  variables  such  as  verbal,  quantita¬ 
tive,  and  analytical  skills  is  analogous  to  what  has  been  done  here  with 
respect  to  aircraft  quality.  In  fact,  one  of  the  chief  motivators  for 
the  development  of  principal  components  analysis  and  related  techniques 
was  the  intelligence  measurement  problem. 
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scores  restated  into  "IQ-type"  units;  i.e.,  rescaled  to  have  a  mean 
equal  to  100  and  a  standard  deviation  equal  to  10. 


TABLE  2 

RESULTS  OF  THE  PRINCIPAL  COMPONENTS  ANALYSIS  FOR 
"QUALITY,"  IN  IQ-TYPE  UNITS 


Aircraft 

Quality 

index 

F-5E 

1 10 

F- 16 

108 

F- 18 

104 

F-  15 

99 

A-6E 

96 

F- 14 

82 

Variables  Used  to  Measure  "Management" 

One  of  the  important  purposes  in  using  principal  components  analy¬ 
sis  is  to  reduce  the  dimensionality  of  data  so  that  interrelations  can 
be  appreciated  more  readily — perhaps  even  be  shown  graphically.  To  this 
end,  principal  components  analysis  was  also  applied  to  indicators  in¬ 
tended  to  describe  the  way  in  which  aircraft  production  programs  are 
managed. 


1.  These  IQ-type  scores  are  not  perfectly  parallel  to  the  actual  values 
commonly  found  in  the  IQ-measurement  literature.  There,  the  mean  is 
always  set  to  100,  as  has  been  done  here,  but  the  value  of  a  standard 
deviation  lies  in  the  range  of  14.5  -  16  (depending  on  the  test,  etc.). 
For  ease  of  interpretation  (as  well  as  to  compress  the  spread,  recogniz¬ 
ing  the  greater  uncertainty  in  the  underlying  measurements),  the  value 
of  a  standard  deviation  in  the  present  case  has  been  set  at  10,  as 
noted.  Both  the  idea  of  an  IQ-type  score  for  an  aircraft  and  the  use  of 
a  simpler  standard  deviation  are  pure  heuristics;  they  in  no  way  affect 
the  analysis  or  conclusions. 

2.  The  actual  principal  components  analysis  is  provided  in  appendix  C. 
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The  indicators  used  for  the  analysis,  here  called  "program  charac¬ 
teristics,"  included  the  following: 

•  Use  of  a  fly-off  competition  in  choosing  a  contractor 
(FYO) 

•  Buying  full  cost  and  schedule  control  system  information 
(C/SCS) 

•  Use  of  incentive  features  in  contracts  ( ITC) 

«  Use  of  multiyear  contracting  (MYC) 

•  Significant  foreign  military  sales  (FMS) 

•  High  level  of  competition  in  subcontracting  (CIS) 

•  Frequent  planned  model  changes  (PMC) 

•  High  level  of  production  rates  (PRL) 

•  Stability  of  production  rates  (PRS) 

•  High  level  of  manning  in  the  program  office  (MPO). 

Table  3  shows  these  characteristics,  measured  in  a  binary  fashion 
for  simplicity  of  analysis  and  presentation.  Simple  "yes"  and  "no" 
answers  are  in  part  natural,  because  the  values  for  the  program  charac¬ 
teristics  are  so  bimodal  that  they  are  virtually  dichotomous,  and  in 
part  judgmental — with  the  judgment  calls  ranging  from  the  easily  sup¬ 
ported  to  the  semiarbitrary .  The  study  team  felt  that  on  the  whole,  the 
reduction  of  scaled  data  to  binary  form  provides  a  large  gain  in  simpli¬ 
city  of  analysis  and  presentation  with  only  a  minor  loss  of  accuracy. 

The  conversion  to  binary  was  done  using  the  following  criteria. 

FYO,  MYC,  and  PRS  tend  naturally  to  be  binary.  Of  the  other  variables, 


1.  In  general,  program  managers  have  little  or  no  control  over  the  pro¬ 
gram  characteristics  listed.  Basically,  they  define  the  acquisition 
management  environment  within  which  program  managers  operate.  Thus, 
management  as  used  here  does  not  refer  to  the  effectiveness  of  program 
managers  and  their  staffs  in  carrying  out  their  ongoing  work.  At  the 
service  secretary  and  CNO/Chief  of  Staff  levels,  however,  these  program 
characteristics  are  largely,  although  not  entirely,  matters  of  choice. 
Hence,  they  are  thought  of  as  observable  indicators  of  management  at 
these  higher  levels. 

2.  A  further  comparison  of  the  differences  between  the  dichotomized  data 
set  and  the  scaled  data  set,  as  well  as  the  rationale  for  the  criteria 
for  the  d ichotomizat ion ,  are  provided  in  appendix  D. 
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PROGRAM  CHARACTERISTICS 
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program  office 


C/SCS  and  ITC  were  coded  as  1  if  two  or  more  production  contracts  had 
these  features.  FMS  was  converted  to  binary  using  the  number  of  years 
the  aircraft  had  been  sold  to  foreign  military  forces;  three  or  more 
years  was  coded  as  1.  CIS,  PRL,  and  MPO  were  based  on  a  high/low  scale 
with  33  percent,  50  airplanes  per  year,  and  100  people,  respectively, 
marking  the  division  between  high  and  lew.  Finally,  PMC  was  converted 
to  binary  using  the  frequency  of  planned  model  changes;  a  model  change 
in  six  years  or  less  was  considered  frequent. 

A  cursory  review  of  table  3  gives  the  information  in  table  4.  If 
the  program  characteristics  listed  are  considered  positive  indicators  of 
a  beneficially  "active,"  high-level  management,  the  table  provides  a 
rough  idea  of  which  programs  have  operated  in  such  an  environment.  This 
approach,  however,  does  not  account  for  any  differences  in  weighting 
among  the  variables.  For  example,  although  the  F-5E  and  the  E-3  have 
exactly  seven  "yes"  answers,  are  high  production  rates  and  stability  of 
production  equal,  or  is  one  more  important  than  the  other?  Thus,  are 
the  F-5E  and  the  E-3  really  in  the  same  high-level  management  environ¬ 
ment?  To  answer  these  questions,  principal  components  analysis  was 
used . 


TABLE  4 

I 

COUNT  OF  THE  NUMBER  OF  "YES"  ANSWERS  PER  AIRCRAFT  IN  THE 
|  BINARY  PROGRAM  CHARACTERISTICS  TABLE 


Number  of 

Aircraft  "yes’1  answers 


F- 16  10 

F-5E  7 

E-3  7 

F- 15  6 

F- 18  5 

E-2C  2 

P-3C  2 

F- 14  0 

A-6E  0 


Results  of  Principal  Components  Analysis  for  "Management" 

Principal  components  analysis  was  applied  to  the  ten  variables 
showr  in  table  3  for  nine  observations  (F-5E,  F-14,  F— 15,  F— 1 6 ,  F- 1 8 , 
A-6E,  E-2C,  P-3C,  and  E-3).  The  correlation  matrix  values  ranged  from 
-0.10  to  1.00.  These  values  are  not  as  uniformly  high  as  those  found  in 
the  principal  components  analysis  for  quality.  The  coefficients  used  to 
form  the  first  principal  component  ranged  between  0.04  and  0.39  and 
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accounted  for  56  percent  of  the  variance.  Again,  the  latter  two  statis¬ 
tics  are  not  as  strong  as  those  in  the  quality  analysis.  Of  the  origi¬ 
nal  ten  variables,  however,  FMS  and  PRS  had  much  lower  coefficients  than 
the  other  variables.  Further  inspection  made  it  clear  that  the  second 
principal  component  accounted  for  a  substantial  portion  of  the  total 
variance,  21  percent,  with  FMS  and  PRS  contributing  heavily.  The  study 
team  concluded  that  FMS  and  PRS  were  measuring  some  variable  other  than 
management.  Upon  reflection,  this  conclusion  was  not  surprising  since 
both  FMS  and  PRS,  although  part  of  the  overall  management  environment, 
are  largely  controlled  by  groups  outside  the  services'  acquisition  pro¬ 
cesses.  In  the  case  of  FMS,  the  contractor,  the  Defense  Security  Assis¬ 
tance  Agency  serving  the  Secretary  of  Defense,  and  the  State  Department 
have  dominant  influences.  In  the  case  of  stability  of  production  (PRS), 
Congressional  funding  levels  have  a  major  influence. 

Because  they  appeared  to  be  measuring  another  concept,  FMS  and  PRS 
were  removed  from  the  list  of  variables.  The  principal  components 

analysis  was  then  recalculated  using  the  remaining  eight  binary  indica¬ 
tors  and  the  same  nine  observations.  The  new  coefficient  matrix  values 
ranged  between  0.32  and  1.00,  and  the  first  principal  component  coeffi¬ 
cients  were  now  more  closely  balanced  (between  0.24  and  0.40)  and  had  an 
R1 2  value  of  69  percent  (i.e.,  they  now  accounted  for  69  percent  of  the 
total  variance). 

The  statistical  significance  of  this  R2  was  checked,  using  a  numer¬ 
ically  generated  sampling  distribution  of  R2  values,  under  the  null 
hypothesis  that  the  underlying  binary  variables  were  uncorrelated.  The 
observed  R2  of  69  percent  was  highly  significant — eight  standard  devia¬ 
tions  above  the  mean  of  the  null  hypothesis  sampling  distribution  (see 
figure  1). 

The  first  principal  component  scores  for  each  of  the  nine  aircraft, 
interpreted  as  "management,"  are  shown  in  table  5.  As  these  scores  have 
no  natural  units,  they  have  been  restated  in  IQ-type  units  for  ease  of 
interpretation;  that  is,  they  have  been  rescaled  to  have  a  mean  equal  to 
100  and  a  standard  deviation  equal  to  10. 


1.  Although  FY0  and  MYC  were  among  the  lowest  coefficients  in  the  first 

principal  component  and  weighed  heavily  in  the  second  principal  compo¬ 
nent  in  the  second  analysis,  they  were  not  removed  as  FMS  and  PRS  had  • 

been  as  there  was  no  sound  reason  for  doing  so.  It  appeared  that  their 

relatively  low  coefficients  were  due  to  their  sparse  presence  (i.e., 

both  FY0  and  MYC  had  only  one  "yes"  in  the  data  set).  This  hypothesis 

was  confirmed  in  the  scaled  data  results  where  the  coefficients  for  FY0  * 

and  MYC  were  among  the  highest.  (See  appendix  D). 

2.  For  a  closer  look  at  these  calculations  and  results,  see  appendix  C. 
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Frequency  of  occurrence 
(percent) 


if 


R2  (percentage  of  variance) 


FIG.  1 :  SAMPLING  DISTRIBUTION  OF  R2  VALUES  COMPARED  TO  R2  VALUE 
CALCULATED  FOR  FIRST  PRINCIPAL  COMPONENT 


TABLE  5 

RESULTS  OF  THE  PRINCIPAL  COMPONENTS  ANALYSIS  FOR 
"MANAGEMENT,"  IN  IQ-TYPE  UNITS 


Aircraft 

Management 

index 

F-1 6 

115 

F-15 

109 

F-5E 

109 

E-3 

106 

F- 1 8 

99 

A-6E 

90 

E-2C 

90 

F-14 

90 

P-3C 

90 

* 
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Relations  Between  Sustaining  Engineering  Usage,  Quality,  and  Management 


As  described  in  the  preceding  sections,  the  dimensions  of  sustain¬ 
ing  engineering  (SE)  usage,  quality,  and  management  have  now  each  been 
reduced  to  single  variables;  thus,  their  relations  can  be  more  readily 
analyzed  and  understood.  Because  a  reliable  quality  score  for  the  E-2C 
is  not  available,  and  because  the  missions  and  mode  of  operation  of  the 
P-3C  and  E-3  are  quite  different  from  those  of  the  other  aircraft,  these 
three  aircraft  have  been  dropped  from  the  comparisons.  The  reduction 
also  permits  analysis  of  a  more  homogeneous  group. 

In  table  6,  the  IQ-type  scores  for  quality  and  management  for  the 
reduced  sample  of  aircraft  are  displayed  together  with  the  SE  usage  mea¬ 
sure.  Although  some  variation  is  seen,  all  of  the  aircraft  show  a  posi¬ 
tive  relation  between  quality  and  management.  No  obvious  relations 
appear  between  management  and  overall  SE  usage,  or  quality  and  overall 
SE  usage. 


TABLE  6 


COMPARING 

"QUALITY"  AND 

"MANAGEMENT"  IN 

IQ-TYPE  UNITS 

AND 

SE  USAGE 

Qual ity 

Management 

SE  usage 

Aircraft 

index 

index 

index 

F-5E 

110 

109 

0.83 

F  —  1 6 

108 

115 

1 .06 

F-18 

104 

99 

1.10 

F— 1 5 

99 

109 

i  .04 

A-6E 

96 

90 

0.82 

F — 1 4 

82 

90 

1  .04 

These  relations  are  more  readily  interpreted  when  presented  graphi¬ 
cally.  Figure  2  shows  the  scores  of  quality  compared  with  the  scores  of 
management.  Quality  and  management  appear  to  have  a  fairly  strong  posi¬ 
tive  correlation  («2  =  0.58  with  a  T  value  of  2.4).  In  this  light,  a 
plausible  hypothesis  is  that  high  scoring  management  should  be  able  to 
economize  on  SE  usage;  thus,  the  two  would  display  a  clear,  negative 
relationship.  Figure  3,  however,  shows  an  essentially  random  pattern, 
without  any  apparent  correlation  between  management  and  SE  usage. 

Figure  4,  displaying  the  scores  of  quality  compared  with  overall  SE 
usage,  also  shows  an  essentially  random  pattern. 
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SE  usage  Quality 


80  85  90  95  100  105  110  115 


Quality 

FIG.  4:  AIRCRAFT  SE  USAGE  COMPARED  WITH  "QUALITY" 


One  possible  explanation  for  the  lack  of  correlation  between  over¬ 
all  SE  usage  and  quality  or  management  is  the  lack  of  an  obviously  "cor¬ 
rect"  and  intuitively  satisfying  way  of  scaling  SE  usage.  Overall  SE 
usage,  as  presented  earlier,  is  the  total  actual  SE  hours  over  the  air¬ 
craft's  career  (or  the  years  for  which  SE  data  were  available)  divided 
by  the  total  "target"  SE  hours  over  the  same  period,  with  values  greater 
than  1  representing,  on  average,  greater  use  of  SE  than  the  estimated 
"target"  and  values  less  than  1  representing,  on  average,  lesser  use  of 
SE  than  the  estimated  "target."  These  differences  are  descriptive 
ratner  than  normative.  For  example,  if  there  were  a  widespread  tendency 
to  use  "too  much"  SE,  then  more  SE  use  than  the  "target"  value  would  be 
"bad"  and  less  use  would  be  "good."  On  the  other  hand,  if  average 
industry  practice  is  "about  right,"  then  the  measure  of  interest  becomes 
the  amount  by  which  the  SE  actual/SE  target  score  departs  from  1 .  This 
departure  can  be  thought  of  as  either  the  algebraic  difference  from  the 
norm  of  1  or  as  the  absolute  difference.  Table  7  shows  these  ways  to 
measure  SE  usage  for  the  various  aircraft  in  the  study's  sample.  Sta¬ 
tistical  analysis  showed,  however,  that  none  of  the  SE  measures  in 
table  7  correlate  with  either  management  or  quality. 

Management,  as  defined  by  program  characteristics,  describes  a 
"management  environment"  created  by  choices  at  the  highest  levels  in  the 
Navy  and  Air  Force.  SE  usage,  however,  is  determined  almost  entirely  by 
program  offices  (and  Plant  Representative  Offices).  Because  a  positive 
correlation  between  high-level  management  and  program  office  management 
does  not  necessarily  have  to  exist,  the  high-level  management  that 
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appears  to  exert  pressure  on  contractors  to  produce  a  high  quality  air¬ 
craft  do^s  not  necessarily  influence  how  much  or  little  SE  is  used 
overall.  ' 


TABLE  7 


Aircraft 

POSSIBLE 

Ratio  of  overall 

SE  actual  to  SE 
target 
(1) 

MEASURES  OF  SE  USAGE 

Algebraic  difference 
between  column  1  and 
and  ratio  of  1.0 
(2) 

Absolute  difference 
between  column  1 
and  ratio  of  1.0 
(3) 

F-5E 

0.83 

-0.  17 

0.17 

F-14A 

1 .04 

0.04 

0.04 

F-15A/B 

1 .04 

0.04 

0.04 

+  C/D 

F-16A/B 

1 .06 

0.06 

0.06 

+  C/D 

F-18A/B 

1 .10 

0.  10 

0.10 

+  C/D 

A-6E 

0.82 

-0.  18 

0.18 

P-3C 

1 .06 

0.06 

0.06 

E-2C 

1 .23 

0.23 

0.23 

E-3 

1  .00 

0.00 

0.00 

The  apparent  lack  of  correlation  between  quality  and  overall  SE 
usage  led  the  study  team  to  revisit  the  basic  hypothesis  that  SE  invest¬ 
ments  in  particular  time  periods  should  result  in  changes  in  quality  in 
related  time  periods. 

In  order  to  test  this  hypothesis,  measurements  of  changes  in  qual¬ 
ity  and  of  time-specific  SE  investments  were  necessary.  Quality  mea¬ 
sures  for  operating  years  two,  four,  and  six  were  calculated  using  prin¬ 
cipal  components  analysis.  The  aircraft  sample  was  restricted  to  those 
aircraft  for  which  both  quality  and  SE  data  for  the  actual  first  six 
operating  years  were  available.  These  six  aircraft  were  the  F— 1 4 ,  F- 1 5 , 
F- 1 8 ,  F-5E,  P-3C,  and  E-3.  (Unfortunately,  SE  data  for  the  first  four 
years  of  the  F— 1 6 ,  one  of  the  most  interesting  aircraft  in  the  study, 
were  not  available.) 

The  variables  used  to  calculate  quality  for  each  year  were  mean 
flight  hours  between  failures  (MFHBF)  in  that  year,  mean  maintenance 


1.  Time  constraints  precluded  trying  to  develop  a  single  measure  of  man¬ 
agement  at  the  program  office  level.  Such  an  attempt  appears  feasible 
and  could  yield  some  useful  insights  on  more  effective  staffing  and  pro¬ 
cedures  at  that  level. 
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man-hours  per  flight  hour  (MMMHFH)  in  that  year,  and  Class  A  and  B  mis¬ 
hap  rates  at  the  appropriate  flight  hours  for  each  aircraft  in  that 
year.  The  change  in  quality  was  then  simply  the  difference  in  the 
quality  scores  for  each  aircraft  between  years  two  and  four,  four  and 
six,  and  two  and  six.  (Figure  5  shows  the  change  in  quality  between 
years  two  and  six.)  For  this  part  of  the  analysis,  the  quality  scores 
use  only  the  three  variables  noted  above  and  have  not  been  converted  to 
IQ-type  scores.  Cumulative  SE  usage  scores  for  the  related  time  periods 
were  calculated  using  the  method  described  earlier. 


Operating  years 


FIG.  5:  CHANGE  IN  QUALITY:  YEAR  TWO  vs.  YEAR  SIX 


Two  other  factors  were  also  taken  into  consideration.  First,  it 
was  necessary  to  distinguish  between  completely  new  aircraft  and  those 
representing  model  changes  in  an  existing  aircraft  (the  F-18  versus  the 
P-3C,  for  example).  The  effect  of  the  first  year's  SE  investment  was 
also  included  on  the  hypothesis  that  it  represents,  in  effect,  the 
effort  needed  to  remedy  deficiencies  in  the  original  design  engineering 
or  systems  engineering.  Thus,  the  equation  takes  the  form: 


bQj,  £  -  Bq  +  B^ASE^  rp_ ,  +  B2NEW  *  B-^SE]  , 


1.  For  details  of  this  principal  components  analysis,  see  appendix  E. 

2.  See  page  2  for  the  definitions  of  these  forms  of  engineering. 
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where 


hQT  =  change  in  quality  between  t  and  T 

A SBt  =  actual  SE/target  SE  for  the  period  t  to  t- 1 

new  =  completely  new  aircraft  versus  a  model  change 
5£1  -  SE  in  the  first  product io!  year. 

This  regression  equation  was  used  to  test  the  hypothesized  rela¬ 
tions.  Because  the  sample  consisted  of  only  six  observations,  the 
regression  was  reestimated  for  a  pooled  sample  of  the  six  aircraft  over 
two  time  periods:  between  year  two  and  year  four,  and  between  year  four 
and  year  six.  To  test  the  acceptability  of  pooling,  an  f-test  was  used. 
The  null  hypothesis  that  the  two  data  sets  came  from  the  same  population 
could  not  be  rejected,  so  the  two  data  sets  were  pooled  to  create  a  set 
of  twelve  observations.  Results  are  shown  in  table  8. 


TABLE  8 

A  REGRESSION  BETWEEN  A  CHANGE  IN  QUALITY  AND  SE  INVESTMENTS 

Dependent  variable:  two-year  change  in  quality 


Explanatory 

variable 

R2  -  .53 
sse  -  1.94 

Estimated 

coefficient 

r-value 

Intercept 

-3.7 

Delta  SE 

4.3 

2.4 

SE  -  first  year 

-1.5 

-1.8 

NEW 

1.5 

2.7 

2 

The  R  for  the  pooled  data  regressions  is  not  significant  at  the 
5-percent  level,  but  is  significant  at  the  10-percent  level.  The  Delta 
SE  investment  variable  is  significant  at  the  5-percent  level,  as  is  the 
new  aircraft  variable. 

All  in  all,  these  results  provide  modest  support  for  the  hypothesis 
that  SE  investments  generate  a  discernible  improvement  in  aircraft 


1.  See  appendix  F  for  a  more  complete  justification  for  using  the  pooled 
data . 
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quality.  This  conclusion,  obviously,  is  subject  to  a  number  of  caveats. 

For  example,  only  one  specification  has  been  considered  here  in  detail, 

and  no  investigation  has  been  undertaken  of  possible  correlation  among  * 

the  error  terms  from  period  to  period.  Thus,  the  results  at  this  stage 

should  be  regarded  only  as  suggestive. 

The  magnitudes  of  the  coefficients  may,  however,  provide  a  rough 
idea  about  the  role  of  SE  in  maintaining  aircraft  quality,  at  least  on  a 
relative  basis . 1 

The  intercept  suggests  that  in  the  absence  of  any  SE  expenditures, 

"aging"  would  lead  to  a  decline  in  quality  of  3.7  units  over  a  two-year 
period.  Attaching  any  value  to  the  intercept  is,  however,  highly  con¬ 
jectural;  if  nothing  else,  there  are  no  actual  observations  in  the  sam¬ 
ple  of  an  aircraft  with  SE  investments  anywhere  near  zero. 

One  interpretation  of  the  coefficients  would  be  that  SE  expendi¬ 
tures  near  the  "target"  value  offset  much  of  the  effect  of  "aging."  A 
typical  value  for  Delta  SE  and  SE  -  first  year  might  be  1  (SE  actual  = 

SE  target) .  Using  this  value  for  both  SE  variables  and  inserting  0  for 
NEW  yields  an  estimate  of  the  change  in  quality  score  of  -0.9  units--or 
roughly  half  of  a  standard  deviation. 

Thus,  the  application  of  typical  amounts  of  SE  may  offset  what 
otherwise  might  be  the  very  large  effects  of  "aging."  This  suggests 
that  deep  cuts  in  SE  could  be  quite  risky.  On  the  other  hand,  the  coef¬ 
ficient  on  Delta  SE  also  suggests  that  making  major  gains  in  quality 
through  SE  investments  could  be  very  difficult.  For  illustrative  pur¬ 
poses,  consider  an  aircraft  whose  quality  is  to  be  improved  from  one 
standard  deviation  below  the  mean  up  to  the  mean;  i.e.,  an  increase  of 
1.6  units  of  quality.  The  coefficient  on  Delta  SE  suggests  that  this 
would  require  a  change  in  the  SE  usage  score,  for  example,  from  the  tar¬ 
get  value  to  37  percent  above  it.  This  is  a  large  movement  in  SE  usage, 
covering  approximately  the  entire  range  in  the  study  sample.  These 
findings  are  consistent,  however,  with  the  study  team's  intuition  that 
the  initial  design  engineering  for  an  aircraft  exerts  a  strong  and  per¬ 
vasive  effect  and  that  basic  design  problems  cannot  be  readily  overcome 
with  SE. 

The  above  results  were  for  the  sample  that  pooled  the  two  periods, 
years  two  through  four  and  years  four  through  six.  Although  the  two 


1.  As  outlined  earlier,  and  as  described  in  more  detail  in  CRM  87-89, 
the  principal  component  score  for  each  aircraft  for  each  point  in  time 
represents  a  relative  measure  of  quality  rather  than  an  absolute  mea¬ 
sure.  To  assess  changes  in  absolute  terms,  it  would  be  preferable  to 
return  to  the  individual  indicators  of  quality,  such  as  MFHBF. 
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samples  were  not  significantly  different  by  an  F- test,  the  coefficients 
do  differ  somewhat,  but  in  intuitive  ways.  Some  of  the  coefficients  in 
the  separate  regressions  and  their  7  values  are  shown  in  table  9- 


TABLE  9 

RESULTS  OF  REGRESSION  ANALYSIS  OF  CHANGES  IN  QUALITY 


Year  two 
through 
year  four 

Year  four 
through 
year  six 

B 

7 

B 

7 

ss2_3 

7.5 

5.5 

5£4-5 

7.5 

2.3 

NEW 

3.2 

6.8 

NEW 

1.0 

1.3 

SE  -j 

-4.4 

-5.5 

S£1 

-0.3 

-0.3 

The  drop  in  7  values  and  coefficients  for  NEW  and  SE^  between  the 
two  time  periods  suggests  that  new  and  se ^  are  less  important  as  the 
time  period  moves  further  away  from  the  first  year;  it  seems  fair  to 
speculate  that  as  time  progressed  they  would  eventually  drop  from  the 
regression . 

In  fact,  the  first  of  the  two  regressions  seems  generally  stronger 
than  the  second  regression  or  the  pooled  data  regression:  in  the  first 
regression,  despite  the  tiny  sample  size,  all  the  7  values  are  signifi¬ 
cant  at  the  5-percent  level. 

Taken  together,  the  three  regressions  give  the  impression  of  a  rea¬ 
sonable  relation  between  quality  improvement  and  the  flow  of  SE  invest¬ 
ments  over  the  appropriate  period.  This  relation  is  summarized  graphi¬ 
cally  in  figure  6.  The  vertical  axis  represents  the  quality  improvement 
between  years  two  and  six.  The  horizontal  axis  shows  the  flow  of  SE 
investments  between  years  two  and  five.  Both  variables  are  "corrected" 
for  SE  in  the  first  year  and  the  NEW  variable.  Thus,  the  plot  is 
actually  a  comparison  of  residuals,  one  from  a  regression  with  Delta  SE 
as  the  dependent  variable,  one  with  the  change  in  quality  as  the  depen¬ 
dent  variable.  These  corrections  are  important  in  explaining  why  the 
data  line  up  as  neatly  as  they  do;  the  raw  data  provide  a  much  more 
random  impression. 


1.  See  appendix  F  for  further  details. 
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Change  in  quality  (AQ) 
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SE  investments 


FIG.  6:  SE  INVESTMENTS  FROM  YEAR  TWO  THROUGH  YEAR  FIVE  COMPARED 
WITH  CHANGE  IN  QUALITY  BETWEEN  YEAR  TWO  AND  YEAR  SIX 
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CONCLUSIONS 


The  fact  that  management  and  quality  are  positively  correlated  sup¬ 
ports  the  hypothesis  that  a  strong,  high-level  management  environment 
raises  the  likelihood  that  a  good  quality  aircraft  will  result.  That 
management  and  SE  usage  do  not  appear  to  be  correlated  is  anti-intuitive 
in  that  the  same  management  that  correlates  with  quality  aircraft  could 
be  expected  to  carefully  use  SE.  The  high-level  management  that  suc¬ 
ceeds  in  having  contractors  produce  quality  aircraft,  however,  is  not 
necessarily  correlated  with  the  program  office  management  that  largely 
controls  SE  usage. 

Finally,  although  quality  and  overall  SE  usage  are  not  directly 
correlated,  a  reasonably  suggestive  positive  correlation  exists  between 
SE  investments  in  a  particular  time  period  and  changes  in  quality 
related  to  that  time  period. 
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COMPONENTS  OF  SUSTAINING  ENGINEERING 


1.  Participating  in  laying  out  and  modifying  production  processes. 

2.  Resolving  problems  of  systems  integration. 

3.  Correcting  production  problems  identified  by: 

Production  supervisors  and  quality  assurance  personnel 
-  Contractor  and  service  flight  tests  (flight  "crabs"). 

4.  Undertaking  engineering  investigations  resulting  from  fleet- 
identified  problems: 

Accidents/ incidents 

Unsatisfactory  reports  by  squadron  pilots  (design-related 

problems  rather  than  simple  malfunctions). 

5.  Preparing  all  engineering  change  proposals  (ECPs). 

6.  Preparing  detailed  drawings/specs  for  approved  correction  of 
defects  (COD)  ECPs. 

7.  Supervising  implementation  of  COD  ECP  work. 

8.  Conducting  engineering  investigations  to  enhance  producibil ity . 

9.  Initiating  and  evaluating  value  engineering  proposals. 

10.  Conducting  engineering  efforts  to  ensure  that  reliability  and 
maintainability  meet  contract  specification  or  desired  levels. 

11.  Providing  engineering  input  to  configuration  control. 

12.  Updating  drawings,  rewriting  specs,  providing  engineering  input 
tech  pubs,  manuals,  etc. 

13.  Maintaining  contractual  engineering  documentation. 

14.  Overseeing  supplier  operations  and  products. 

15.  Maintaining  liaison  with  service  acquisition  managers  and  their 
associated  field  activities. 


16.  Managing  in-house  sustaining  engineering  efforts. 
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ANNOTATED  BIBLIOGRAPHY  OF  RESEARCH  MEMORANDUMS 
IN  THE  SERIES  ON  SUSTAINING  ENGINEERING 


CNA  Research  Memorandum  87-40,  Managing  Sustaining  Engineering  in  Weapon 
Systems:  Phase  1,  by  John  E.  Keller  et  al . ,  Mar  1987. 

This  research  memorandum  (RM)  explains  the  origin  of  the  VCNO's 
tasking  of  CNA  to  study  sustaining  engineering.  It  describes  the  analy¬ 
tic  and  procedural  approach  to  the  problem  as  originally  conceived  and 
reports  some  of  the  earlv  substantive  findings.  The  most  important  of 
these  was  that  the  Navy  does  not  know,  nor  can  it  currently  find  out  in 
a  comprehensive  and  systematic  way,  the  specific  services  it  receives  as 
a  result  of  its  annual  expenditure  of  approximately  $1  billion  on  sus¬ 
taining  engineering  for  aircraft  and  missiles.  (When  these  preliminary 
results  were  briefed  to  the  VCNO,  the  original  tasking  was  revised  and 
expanded.)  The  general  contents  of  subsequent  documents  in  this  series 
are  also  outlined. 

CNA  Research  Memorandum  87-69,  Contractors'  Arrangements  for  Estimating, 
Controlling,  and  Recording  Sustaining  Engineering,  Volume  I,  by  John  E. 
Keller  et  al.,  May  1987. 

The  management  of  sustaining  engineering  (SE)  is  examined  in  detail 
for  selected  weapon  systems  at  six  aircraft  and  missile  contractors' 
plants.  The  RM  discusses  various  aspects  of  weapon  system  engineering 
and  trends  in  the  use  of  engineering  man-hours  devoted  to  SE  during 
full-scale  production  and  subsequent  deployment  of  the  systems.  Methods 
of  estimating,  budgeting,  controlling,  and  accounting  for  SE  hours  and 
costs  are  discussed. 

This  volume  also  summarizes  and  interprets  the  formal  presentations 
as  well  as  the  less  formal  discussions  with  management,  engineering,  and 
contracting  personnel  at  the  six  contractors'  plants  visited  by  the  CNA 
study  team.  In  addition,  summaries  of  discussions  with  personnel  from 
the  Air  Force  F- 1 6  System  Program  Office  (SPO)  and  Plant  Representative 
Offices  (PR Os)  are  included. 

CRM  87-69  is  supplemented  by  Volume  II  (CNA  Research  Memorandum 
87-70,  May  1987),  which  is  made  up  of  the  CNA  questionnaire  on  SE  and 
the  contractors'  complete,  formal  responses. 

CNA  Research  Memorandum  87-70,  Contractors '  Arrangements  for  Estimating, 
Controlling,  and  Recording  Sustaining  Engineering,  Volume  II,  by  John  E. 
Keller  et  al . ,  May  1987. 

This  RM  (Volume  II  of  CRM  87-69)  provides  the  texts  of  materials 
prepared  by  selected  aerospace  contractors  for  a  CNA  study  team 
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investigating  sustaining  engineering  on  Wavy  (and  Air  Force)  weapon  sys¬ 
tems.  The  supplementary  material  in  this  research  memorandum  provides 
more  detailed  information  than  is  available  in  CRM  87-69  (Volume  I)  and 
affords  the  critical  reader  an  opportunity  to  check  the  summary  findings 
and  interpretations  of  that  volume. 

CNA  Research  Memorandum  87-xxxx,  Estimating  Sustaining  Engineering 
"Requirements"  for  Military  Aircraft,  by  James  M.  Jondrow  et  al., 
forthcoming. 

A  statistical  analysis  of  sustaining  engineering  (SE)  for  military 
aircraft  is  given  in  this  KM.  The  data  used  are  actual  SE  hours,  over 
various  annual  periods  between  1970  and  1986,  for  a  pooled  sample  of 
nine  aircraft:  F-5E,  F-14,  F-15,  F- 16 ,  F-18,  A-6E,  P-3C,  E-2C,  and  E-3. 
Regression-based  estimates  of  SE  hours  were  derived  as  a  function  of 
production  lot  size,  cumulative  production  quantities,  model  changes, 
airframe  weight  (to  represent  scale),  density  of  the  aircraft  (to  repre¬ 
sent  manufacturing  complexity),  and  maximum  mach  number  (to  represent 
design  complexity).  To  allow  for  contractor  lags  in  adjusting  to 
changes  in  rates  of  production,  SE  hours  for  the  previous  year  were  also 
included.  The  interpretation  of  the  estimating  equation  is  as  a  statis¬ 
tical  means  of  codifying  currenc  industry  practice,  not  as  a  means  of 
determining  "requirements." 

p 

The  estimating  equation  had  an  R  of  91  percent,  and  all  of  the 
variables  had  statistically  highly  significant  coeff icients--even  the 
model  change  variable  which  was  plagued  by  data  problems. 

Estimated  SE  hours  were  compared  with  actual  SE  hours  for  the  nine 
aircraft  over  the  periods  for  which  actual  data  were  available  to  obtain 
a  measure  of  relative  SE  usage.  This  comparison  revealed  both  one- 
period  differences  between  actual  and  target  values  and  extended-period 
differences,  usually  toward  the  end  of  the  production  period,  during 
which  the  actual  values  appeared  to  be  driven  by  factors  other  than  the 
variables  in  the  model . 

CNA  Research  Memorandum  87-xxxx,  Measures  of  Effectiveness  Related  to 
Aircraft  Reliability,  Ma inta inab i 1 ity,  and  Safety  and  Believed  to  be 
Influenced  by  Sustaining  Engineering ,  by  John  E.  Keller  et  al., 
forthcoming. 

Data  are  provided  in  both  tabular  and  graphical  form  for  measures 
of  effectiveness  related  to  aircraft  reliability,  maintainability,  and 
safety  and  that  are  believed  to  be  influenced  by  the  application  of  con¬ 
tractors'  sustaining  engineering  effort.  In  addition  to  the  data  them¬ 
selves,  definitions,  commentary,  and  interpretation  are  provided. 

The  measures  of  effectiveness  (MOEs)  for  which  data  are  provided 
include:  average  number  of  flights  required  for  acceptance;  average 
number  of  discrepancies  per  acceptance  flight;  mean  flight  hours  between 
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failures  (MFHBF);  mean  maintenance  man-hours  per  flight  hour  (MMMHFH); 
mean  maintenance  man-hours  per  maintenance  action  (MMMHMA);  ground  and 
in-flight  aborts  per  100  sorties;  and  combined  Class  A  and  B  mishaps  per 
100,000  flight  hours.  Data  for  the  MOEs ,  with  some  exceptions,  are 
available  for  the  following  aircraft:  F-5E;  F-14A;  F-15A/B  and  C/D; 
F-16A/B  and  C/D;  F-18A/B;  A-6E;  P-3C;  E-2C;  and  E-3A/B/C.  The  data  gen¬ 
erally  cover  the  period  from  the  type-model-series's  first  full  opera¬ 
tional  year  through  1985  or  1 986 .  F - 1 4 A  and  F-18A/B  data  are  also 
available,  by  block.  Navy  and  Air  Force  data  have  been  made  very  nearly 
fully  consistent.  With  the  exception  of  acceptance  testing  histories, 
the  data  relate  only  to  those  parts  of  the  aircraft  (work  unit  or  system 
codes)  for  which  the  prime  contractor  and  his  sustaining  engineering  are 
responsible. 

CNA  Research  Memorandum  87-89,  Measures  of  Aircraft  Reliability,  Main¬ 
tainability  ,  and  Safety:  Do  They  Measure  "System  Quality"?  by  Peter  E. 
Hilsenrath  et  al . ,  Jun  1987. 

This  research  memorandum  applies  principal  components  analysis  to 
nine  indicators  of  aircraft  reliability,  maintainability,  and  safety. 

The  indicators  are  standard  readiness  measures,  such  as  mean  time 
between  failure  at  different  points  in  the  aircraft's  career.  The  data 
are  adjusted  to  include  only  the  airframe,  the  portion  of  the  aircraft 
the  prime  contractor  is  responsible  for.  The  assumption  behind  the 
analysis  is  that  the  observable  variables  are  indicators  of  an  under¬ 
lying  unobservable  variable,  "airframe  quality."  This  principal  compo¬ 
nents  analysis  provides  a  quality  "score"  for  each  aircraft.  Five  of 
the  six  aircraft  analyzed  fall  within  one  standard  deviation  of  the 
average  score;  only  one  falls  below.  The  scores  for  the  first  principal 
component  account  for  85  percent  of  the  variation  in  the  original  indi¬ 
cators,  which  is  highly  significant  in  a  statistical  sense.  This  find¬ 
ing  supports  the  initial  hypothesis,  that  there  is  such  a  thing  as 
"quality,"  which  is  a  major  source  of  the  variation  across  aircraft  in 
the  nine  indicators. 

CNA  Research  Memorandum  87-9 2,  Relations  Between  Sustaining  Engineering, 
Methods  of  Management,  and  System  Quality,  by  Barbara  L.  Tuxbury  et  al .  , 
Dec  1987. 

One  of  the  major  elements  in  the  Vice  Chief  of  Naval  Operations 
(VCN0)  tasking  for  the  sustaining  engineering  (SE)  study  was  to  examine 
the  relations,  if  any,  between  SE  usage,  methods  of  management,  and  mea¬ 
sures  of  effectiveness  related  to  reliability,  maintainability,  and 
safety.  Another  research  memorandum  (RM)  in  this  series  estimates  rela¬ 
tive  SE  usage  among  a  large  sample  of  Navy  and  Air  Force  aircraft. 

Still  another  RM  in  this  series  examines  measures  of  effectiveness  for 
aircraft  reliability,  maintainability,  and  safety.  That  RM  described 
how  it  was  found  possibLe,  using  principal  components  analysis,  to  com¬ 
bine  multiple  measures  of  effectiveness  into  a  single  measure  of  air¬ 
craft  "quality." 
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In  order  to  complete  the  comparisons  requested  in  the  tasking,  a 
measure  of  management  is  necessary.  The  same  technique  of  principal 
components  analysis  has  been  employed  to  combine  multiple  program  char¬ 
acteristics  into  a  single  measure  of  "management."  (These  program  char¬ 
acteristics  describe  the  type  of  "management  environment,"  created  by 
choices  at  the  service  secretary  and  CNO/Chief  of  Staff  leveis,  within 
which  a  program  office  works.  Hence,  this  kind  of  "management"  is  not 
an  indicator  of  the  program  office's  effectiveness.)  The  analysis  found 
that  the  first  principal  component  accounted  for  a  statistically  signif¬ 
icant  percentage  of  the  variation  in  the  program  characteristics,  thus 
supporting  the  idea  of  a  single  underlying  variable,  "management." 

The  three  single  measures — relative  SE  usage,  "quality,”  and 
"management" — were  then  used  to  seek  out  relations,  if  any.  The  analy¬ 
sis  revealed  that  quality  and  management  show  a  significant  positive 
correlation.  Management  (as  defined  by  program  characteristics)  and 
overall  relative  SE  usage  show  no  obvious  correlation.  Although  quality 
and  overall  relative  SE  usage  show  no  apparent  correlation,  SE  invest¬ 
ments  in  a  particular  time  period  and  changes  in  quality  related  to  that 
time  period  do  show  a  positive  correlation. 

CNA  Research  Memorandum  87-xxxx,  The  Impact  of  Sustaining  Engineering 
and  of  Aging  on  Aircraft  Reliability,  Maintainability,  and  Safety,  by 
Barbara  L.  Tuxbury  et  al . ,  forthcoming. 

The  ways  in  which  military  aircraft  (actually  airframes)  "age,"  or 
change  in  "quality"  with  time  or  use  are  explored  in  this  RM.  Quality 
includes  measures  of  reliability,  maintainability,  and  safety.  Because 
sustaining  engineering  (SE)  is  used  to  support  both  the  production  pro¬ 
cess  and  operational  aircraft,  there  should  be  evidence  of  a  favorable 
impact  on  the  quality  of  aircraft  newly  off  the  production  line  and  on 
quality  measures,  over  time,  for  operational  aircraft.  In  fact,  relia¬ 
bility  and  maintainability  for  aircraft  newly  off  the  production  line 
are  clearly  improved,  and  the  improvement  may  be  attributable,  in  part, 
to  SE.  Safety  measures  for  operational  aircraft  are  also  improved,  and 
this  improvement  persists  over  time.  There  is,  however,  a  tendency  for 
the  measures  of  reliability  and  maintainability  to  worsen  over  time-- 
sometimes  dramatically;  and  there  is  some  evidence  of  a  tendency  among 
certain  aircraft  for  successive  and  initially  better  blocks  to  converge 
to  a  common  level  of  quality.  Such  tendencies  must  be  taken  into 
account  in  trying  to  assess  the  effect  of  SE  on  aging.  A  related  find¬ 
ing  of  the  research  is  that  patterns  of  aging  are  obscured  when  aircraft 
are  analyzed  as  a  fleet;  the  continued  introduction  of  new,  better  air¬ 
craft  masks  the  decline  in  quality  of  the  preexisting  inventory. 

During  the  course  of  studying  the  aging  problem,  related  issues 
were  investigated  in  a  preliminary  fashion.  The  issues  included:  the 
use  of  work  unit/system  codes  to  identify  key  contributors  to  low  relia¬ 
bility  and  maintainability  (thus  providing  the  basis  for  more  effec¬ 
tively  directing  future  SE  efforts),  aircraft  procurement  requirements 
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to  ensure  having  a  specific  number  of  aircraft  that  could  even  become 
mission  capable,  and  the  long-term  cost-effectiveness  of  buying  new  air¬ 
craft  versus  remanufacturing  old  aircraft. 

The  final  portion  of  the  RM  outlines  a  number  of  additional  issues 
that  appear  worthy  of  continued  or  fresh  analysis. 

CNA  Research  Memorandum  87-xxxx,  Sustaining  Engineering  Hours  and  Costs, 
Manufacturing  Hours  and  Costs,  Other  Hours  and  Costs,  and  Related  Air¬ 
craft  Production,  by  John  E.  Keller  et  al.,  forthcoming. 

Data  are  provided  in  both  tabular  and  graphical  form  on  sustaining 
engineering  hours  and  costs,  manufacturing  hours  and  costs,  and  other 
contractor  hours  and  costs  related  to  annual  aircraft  production  buys. 

In  addition  to  the  data  themselves,  definitions,  commentary,  and  inter¬ 
pretation  are  provided. 

The  data  are  from  two  main  sources:  the  Contractor  Cost  Data 
Report  (Forms  1921-1  as  required  by  DOD  Instruction  7000.11)  and  a  spe¬ 
cial  CNA-designed  data  format.  Data  from  these  two  sources  could  not  be 
made  congruent  and,  hence,  special  emphasis  was  placed  on  ensuring  maxi¬ 
mum  accuracy  and  consistency  among  contractors  using  the  CNA-designed 
format.  Data  were  generated  for  the  F-5E,  F-14,  F- 1 5 ,  F-16,  F- 1 8 , 

AV-8B,  A-6E,  P-3C,  E-2C,  and  E-3.  In  most  cases,  the  data  covered  the 
full  production  history  of  the  aircraft  through  1986. 

These  original  data  showed  widely  different  levels  of  sr  taining 
engineering  (SE)  hour  use,  both  in  total  and  on  a  per  aircraft  basis. 

The  SE  hour  disparities  created  a  need  to  recognize  differences  in  air¬ 
craft  size,  complexity,  etc.  The  subsequent  analytic  effort  to  deal 
with  these  differences  is  reported  in  the  related  RM,  Estimating 
Sustaining  Engineering  "Requ irements "  in  Military  Aircraft,  which  is 
forthcoming. 
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COMPARISON  OF  PRINCIPAL  COMPONENTS  ANALYSES  USING  BINARY 

AND  SCALED  DATA 


In  order  to  determine  if  dichotomizing  the  values  for  program  char¬ 
acteristics  distorted  the  results  of  the  principal  components  analysis, 
a  comparison  was  made  using  both  scaled  and  dichotomized  variables.  In 
the  original  dichotomization ,  the  scaled  data  were  studied  and  judgments 
were  made  about  reasonable  bases  for  classifying  the  data  into  simple 
"yes"  and  "no"  answers.  (See  table  3  in  the  main  text.)  In  the  case  of 
both  fly-off  competition  and  multiyear  procurements  this  was  simple: 
there  was  only  one  example  of  each.  The  one  example  of  fly-off  competi¬ 
tion  was  between  the  YF— 1 6  and  the  YF-17.  Because  high-level  "manage¬ 
ment"  choices  are  being  evaluated,  only  the  F— 1 6  is  given  credit  for  a 
fly-off  competition,  as  the  Air  Force  instigated  the  competition  that 
led  to  its  choice.  A  heavily  reworked  form  of  the  YF-17  later  became 
the  F— 1 8 ,  but  the  F— 1 8  was  substantially  different  from  the  YF-17  and 
was  not  developed  primarily  as  a  result  of  the  original  competition. 
Thus,  fly-off  competition  was  scored  "no"  for  the  F- 1 8 .  In  either  case, 
however,  this  characteristic  is  naturally  dichotomous.  Similarly,  there 
is  only  one  example  of  multiyear  procurements  (the  F-16),  and  it  has  had 
two  four-year  procurements  of  that  kind.  This  experience  is  neither 
short  term  nor  freakish.  Again,  the  variable  is  naturally  dichotomous. 

Program  office  staffing  is  a  characteristic  so  highly  bimodal  that 
it  was  best  represented  by  a  simple  high-low  score.  A  group  of  program 
offices  has  staff  sizes  between  20  and  50.  No  other  values  are  seen 
until  another  group  emerges  with  staff  sizes  between  roughly  130  and 
500.  There  is  no  question  that  there  are  just  two  groups:  large  and 
small.  (If  pressed,  of  course,  the  large  group  could  be  divided  into 
medium  and  large,  but  the  advantage  to  that  is  not  obvious.  Moreover, 
additional  data  would  be  needed  to  make  the  division  between  medium  and 
large. )  Although  bigger  is  not  necessarily  better  organizationally,  the 
weighting  of  this  characteristic  in  the  principal  components  analysis 
was  quite  heavy.  Furthermore,  the  importance  of  size  pertains  to  staff¬ 
ing  of  the  program  office  only — not  the  organizational  superstructure 
above  it.  The  finding  is  supported,  too,  by  observation  of  many  of  the 
small-size  program  offices;  they  appear  to  be  stretched  very  thin  and 
ari.  fci'ced  into  playing  catch-up  with  the  contractor. 

Incidentally,  this  program  characteristic  illustrates  many  of  the 
"counting"  problems  that  arise  from  trying  to  use  scaled  variables. 
Should  staff  size  be  measured  at  its  minimum,  its  maximum,  or  by  a  time- 
weighted  average?  Should  size  be  related  to  contract  value?  To  produc¬ 
tion  rates?  Should  it  be  lagged  or  not?  Should  rank  or  grade  of  the 
staff  be  taken  into  account?  Would  average  program  office  cost  be  bet¬ 
ter?  Averaged  over  what  period  of  time?  If  program  staffing  and  proce¬ 
dures  were  a  key  issue  (and  they  might  well  be  worth  a  separate  study), 
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then  all  of  the  questions  noted  above  would  be  worth  answering.  How¬ 
ever,  for  the  present  study,  a  simple  distinction  between  large  and 
small  seemed  adequate  and  appropriate. 

Program  characteristics,  such  as  high  and  low  production  rates  and 
frequency  of  planned  model  changes,  admittedly  are  much  more  judgmental. 
Nevertheless,  there  was  some  basis  for  the  judgments.  For  example,  con¬ 
ventional  wisdom  holds  that  four  aircraft  per  month  is  a  minimum  effi¬ 
cient  production  rate;  hence,  annual  buy  sizes  above  or  below  50  were 
chosen  as  being  high  or  low  production  rates.  The  six-year  test  on 
planned  model  changes  does,  indeed,  verge  on  the  arbitrary.  With  the 
exception  of  the  F— 1 8 ,  however,  all  of  the  aircraft  fell  fairly  clearly 
on  one  side  of  that  dividing  line  or  the  other.  For  example,  the  F-14, 
A-6E,  E-2C,  and  P-3C  all  have  been  unchanged  for  a  decade  or  more.  The 
F— 1 8  A/B  became  a  C/D  in  FY  1986,  thus  just  missing  the  six-year  divid¬ 
ing  line.  When  the  VCNO  was  briefed  in  April  1986,  however,  the  FY  1986 
buy  was  advertised  as  a  major  block  upgrade,  not  as  a  model  change,  thus 
weakening  its  claim  as  a  planned  model  change.  (Accepting  major  block 
upgrades  as  equivalent  to  model  changes  opens  a  dismaying  Pandora's 
box. ) 


The  buying  of  full  C/SCS,  the  use  of  incentive  type  contracts,  the 
use  of  competition  in  subcontracting,  the  amount  of  stability  in  produc¬ 
tion,  and  the  significance  of  foreign  military  sales  can  all  be  scaled, 
but  once  again  the  scaling  shows  trends  toward  two  groups  of  data. 

Table  D-1  shows  the  R2  and  T  values  for  regressions  comparing  the  scaled 
values  of  the  data  (as  shown  in  table  D-2)  to  the  binary  measures  for 
the  data  (as  shown  in  table  3  in  the  main  text).  Except  for  foreign 
military  sales  and  competition  in  subcontracting,  the  binary  measures 
appear  to  correspond  well  to  the  scaled  measures. 

In  order  to  see  how  these  differences  might  affect  the  results  of 
the  original  analysis  with  binary  data,  principal  components  were 
calculated  using  scaled  values  for  the  variables.  The  results  of  these 
analyses  are  shown  in  figure  D-1;  they  are  also  shown  graphically  in 
figure  D-2.  The  scaled  variables  provide  a  slightly  greater  spread  in 
the  distribution  of  scores,  but  overall  they  present  a  picture  similar 
to  that  for  the  binary  data. 


1.  The  actual  principal  component  analyses  are  shown  in  tables  D-3  and 
D-4 . 
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TABLE  D-1 


CORRELATION  BETWEEN  BINARY  AND  SCALED  MEASURES 

p 

(  k  =  percent) 


FYO 

R2 

:  100 

T 

= 

* 

CSS 

R2 

- 

12 

T 

= 

4.2 

ITC 

R2 

72 

T 

= 

4.2 

MYC 

R2 

- 

100 

T 

= 

* 

FMS 

R2 

33 

T 

= 

1.8 

CIS 

R2 

- 

53 

T 

= 

2.8 

PMC 

R2 

- 

60 

T 

= 

-3.2 

PRL 

R2 

- 

85 

T 

z 

6.3 

PRS 

R2 

- 

75 

T 

= 

4.6 

MPO 

R2 

- 

62 

T 

= 

3.4 

p 

*  The  T  value  is  undefined  when  r  =  100  percent. 


D-3 


T3 

CD 

CO 

>> 

3 

X 

X 

*a 

X 

CD 

b0 

•rH 

c 

•-H 

CD 

Cl 

L* 

•rH 

X 

X 

CO 

r-H 

3 

E 

03 

CD 

»* 

CL 

S- 

CD 

X 

c 

E 

o 

• 

3 

•rH 

CO 

C 

X 

r-H 

a 

«  • 

CD 

X 

3 

X 

T3 

CO 

*o 

O 

O 

CD 

o 

03 

E 

X 

c. 

• 

bo 

Q. 

4-> 

CO 

Cm 

•rH 

c 

CD 

O 

• 

X 

Cm 

o 

a 

C 

O 

o 

s- 

J- 

o 

>> 

o 

CD 

•H 

X 

X 

L- 

Cm 

X) 

4-) 

3 

• 

03 

E 

o 

T3 

O 

*o 

CD 

3 

3 

<D 

<1) 

>> 

t- 

C 

TJ 

T3 

CO 

CO 

•rH 

03 

O 

•rH 

•rH 

3 

4-) 

4-) 

r-H 

> 

H 

♦rH 

• 

03 

a. 

•rH 

X 

CO 

3 

rH 

>>-l> 

a 

4-> 

E 

•rH 

r—i 

O 

c 

■rH 

a 

E 

CD 

X 

•rH 

r. 

X 

03 

03 

> 

t- 

3 

c 

•rH 

>> 

CO 

o 

x 

>» 

CO  -u> 

X) 

s~ 

Cm 

«■. 

• 

c 

XI 

•rH 

•rH 

03 

QJ 

X 

o 

CD 

X 

TD 

CD 

TD 

Lf\ 

X 

o 

*o 

£- 

CD 

CD 

>> 

CD 

1 

bo 

CD 

O 

Q.  T3 

X 

U- 

3 

CD 

L- 

<X 

E 

•rH 

r-H 

O 

O 

a 

CD 

O 

> 

01 

03 

CD 

X 

>» 

> 

O 

a 

•rH 

X 

L* 

X 

4-) 

O 

4-> 

"O 

O 

X 

X 

CO 

* 

o 

TD 

4-> 

c 

CJ 

0) 

•o 

CD 

c 

o 

c. 

CO 

> 

4-> 

rH 

X3 

o 

>» 

a 

o 

N 

•H 

a 

o 

s- 

■rH 

X 

Cm 

u 

X 

03 

CO 

03 

X 

X 

c 

i - 

3 

a 

“O 

Cm 

X 

(D 

4-> 

c 

03 

3 

0) 

03 

CL 

*"H 

a 

c 

o 

“O 

T3 

X 

CD 

3 

c 

o 

•rH 

O0 

o 

•rH 

o 

O 

Cm 

-*H 

o 

X 

c 

> 

X 

o 

•rH 

CL 

•rH 

•rH 

CD 

CO 

CO 

CO 

3 

X 

T3 

03 

t- 

L* 

"O 

a 

c 

•» 

03 

03 

03 

o 

03 

•rH 

•o 

Cm 

vO 

0) 

CD 

CD 

c 

<D 

O 

CO 

>» 

>> 

>> 

CL  X 

CO 

O 

ON 

c 

u 

3 

Lh 

« — 

H 

H 

H 

r—i 

o 

03 

T3 

CD 

03 

03 

03 

03 

a 

<D 

O 

X 

c 

x 

X 

X 

4-> 

X) 

>» 

u 

E 

•rH 

O 

O 

O 

O 

3 

a. 

3 

X 

4-> 

4-> 

X 

CO 

Cm 

C 

Cm 

O 

X 

Cm 

Cm 

Cm 

Cm 

Cm 

Cm 

Cm 

r-H 

03 

O 

O 

o 

O 

O 

S~ 

03 

03 

X 

a> 

S-. 

3 

CO 

Cl) 

CD 

CD 

CD 

CD 

X 

o 

C 

bO 

bO 

bO 

bo 

bo 

E 

C 

Cm 

03 

03 

03 

03 

03 

3 

•rH 

03 

O 

X 

4-> 

4-> 

X 

X 

c 

03 

C 

C 

c 

c 

c 

X 

S-. 

0) 

CD 

CD 

CD 

CD 

r-H 

t— H 

CO 

CD 

a 

O 

O 

O 

O 

03 

03 

CD 

•  X 

i.s-s-s-t.jj*j3oe 

<D<Doia>a>ooooD 

a,a,a,cua,hE-'J>-2 


(CJ2  O  X)  0)  <1-1  OOJC 


BINARY  DATA 


SCALED  DATA 

9  observations  and  10  variables 


9  observations  and  10  variables 

9  observations  and  10  variables 

R2  =  56 

IQ  Scores 

First  principal 

R2  =  63 

IQ  Scores 

First  principal 

F-16 

116 

component 

F-16 

121 

component 

coefficients 

coefficients 

F-5E 

109 

range  from  .04 

E-3 

106 

range  from  .15 

F-15 

108 

to  .39  Low 

F-5E 

103 

to  .39.  Low 

E-3 

107 

coefficients  for 

F-18 

102 

coefficients  for 

F-18 

100 

FMS  and  PRS. 

F-1S 

101 

FMS  and  PRS. 

P-3C 

91 

Second 

P-3C 

93 

Second 

E-2C 

91 

principal 

E-2C 

92 

principal 

component 

component 

F-14 

90 

coefficients  are 

F-14 

92 

coefficients  are 

A-6E 

90 

heavy  for  FMS 

A-6E 

90 

heavy  for  FMS 

and  PRS. 

and  PRS. 

9  observations  and  8  variables 

9  observations  and  8  variables 

R2  =  69 

IQ  Scores 

First  principal 

R2  =  73 

IQ  Scores 

First  principal 

F-16 

115 

component 

F-16 

122 

component 

coefficients 

coefficients 

F-5E 

109 

range  from  .24 

E-3 

106 

range  from  .30 

F-15 

109 

to  .40. 

F-5E 

103 

to  .40. 

E-3 

106 

F-15 

103 

F-18 

99 

F-18 

101 

P-3C 

90 

P-3C 

93 

F-14 

90 

F-14 

92 

E-2C 

90 

E-2C 

91 

A-6E 

90 

A-6E 

90 

FIG.  D-1:  PRINCIPAL  COMPONENT  ANALYSES  OF  PROGRAM  CHARACTERISTICS 


D-5 


FIG.  D-2:  SCORES  BASED  ON  SCALED  DATA  AND  ON  BINARY  DATA 
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APPENDIX  E 


PRINCIPAL  COMPONENTS  ANALYSES  FOR  QUALITY  IN 
DIFFERENT  TIME  PERIODS 


In  order  to  test  the  hypothesis  that  SE  investments  in  particular 
time  periods  should  result  in  changes  in  quality  in  related  periods, 
measurements  of  changes  in  quality  were  necessary.  Quality  measures  for 
operating  years  two,  four,  and  six  were  calculated  using  principal  com¬ 
ponents  analysis.  The  aircraft  sample  was  restricted  to  those  aircraft 
for  which  both  quality  and  SE  data  were  available  in  operating  years 
two.  four,  and  six.  The  variables  used  to  calculate  quality  for  each 
year  were  MFHBF  in  that  year,  MMMHFH  in  that  year,  and  Class  A  and  B 
mishap  rates  at  the  appropriate  flight  hours  for  each  aircraft  in  that 
year.  Tables  E-1  through  E-3  show  the  actual  principal  components  anal¬ 
yses  and  resulting  quality  scores. 


E-1 


TABLE  E-1 


PRINCIPAL  COMPONENTS  ANALYSIS 

(six  observations  and  three  variables--quality  for  year  two) 


Simple  statistics 

MFHBF2  MMHFH2  ANBMR2 

Mean  2.10  6.17  7.63 

Standard  0.98  4.01  6.29 

deviat ion 


Correlations 


MFHBF2 

MMHFH2 

ANBMR2 

MFHBF2 

1.00 

-0.81 

-0.41 

MMHFH2 

-0.81 

1.00 

0.80 

ANBMR2 

-0.41 

0.80 

1.00 

Eigenvalue  Difference 


Proportion  Cumulative 


First  principal  component  2.36 
Second  principal  component  0.60 
Third  principal  component  0.05 


1.76  0.79  0.79 

0.55  0.20  0.99 

0.01  1.00 


Coeff icients 


First 

Second 

Third 

pr incipai 

principal 

principal 

component 

component 

component 

MFHBF2 

-0.54 

0.70 

0.46 

MMHFH2 

0.64 

-0.01 

0.77 

ANBMR2 

0.54 

0.71 

-0.45 

First 

principal 

Aircraft 

component 

P-3C 

-1  .91 

F-5E 

-1 .49 

E-3 

-0.25 

F- 18 

0.50 

F- 15 

1  .00 

F-  14 

2.15 

E-2 


TABLE  E-2 


PRINCIPAL  COMPONENTS  ANALYSIS 

(six  observations  and  three  variables — quality  for  year  four) 

Simple  statistics 


MFHBF4 

MMHFH4 

ANBMR4 

Mean 

2.72 

5.97 

5.45 

Standard 

1 .  14 

4.92 

4.61 

deviat ion 

Correlations 


MFHBF4 

» 

MMHFH4 

ANBMR4 

MFHBF4 

1.00 

-0.84 

-0.50 

MMHFH4 

-0.84 

1.00 

0.85 

ANBMR4 

-0.50 

0.85 

1.00 

Eigenvalue  Difference  Proportion  Cumulative 


First  principal  component  2.47 
Second  principal  component  0.50 
Third  principal  component  0.03 


1.97  0.82  0.82 

0.47  0.17  0.99 

0.01  1.00 


Coefficients 


First  Second  Third 

principal  principal  principal 
component  component  component 

MFHBF4  -0.55  0.72  0.43 

MMHFH4  0.63  0.01  0.78 

ANBMR4  0.55  0.70  -0.46 


First 
pr incipal 
Aircraft  component 


P-3C 

-1.65 

E-3 

-0.95 

F-5E 

-0.79 

F- 18 

0.10 

F-  15 

0.47 

F-  14 

2.81 

TABLE  E-3 


PRINCIPAL  COMPONENTS  ANALYSIS 

(six  observations  and  three  variables — quality  for  year  six) 


Simple  statistics 


MFHBF6  MMHFH6  ANBMR6 


Mean  2.35  6.98  3.92 

Standard  1.01  5.19  4.47 

deviation 


* 


Correlations 


MFHBF6 

MMHFH6  ANBMR6 

MFHBF6 

1.00 

-0.72  -0.44 

MMHFH6 

-0.72 

1.00  0.93 

ANBMR6 

-0.44 

0.93  1.00 

Ei 

genvalue 

Difference  1 

Proportion 

Cumulative 

First 

principal 

component 

2.41 

1 .83 

0.80 

0.80 

Second 

principal 

component 

0.58 

0.57 

0.19 

0.99 

tv,  <  rcj 

principal 

component 

0.01 

0.01 

1 .00 

Coefficients 


First 
pr incipal 
component 

Second 

principal 

component 

Third 

principal 

component 

MFHBF6 

-0.51 

0.81 

0.30 

MMHFH6 

0.64 

0.  12 

0.76 

ANBMR6 

0.58 

0.57 

-0.58 

First 

pr incipal 


Aircraft 

component 

F-5E 

-1.37 

P-3C 

-0.75 

F- 18 

-0.59 

E-3 

-0.36 

F- 15 

0.06 

F- 14 

3.02 

E-4 


APPENDIX  F 

EXCURSIONS  ON  THE  STATISTICAL  ANALYSIS  OF  THE  RELATION  BETWEEN 
SE  INVESTMENTS  OVER  TIME  AND  CHANGES  IN  QUALITY  FOR 
RELATED  TIfffi  PERIODS 


APPENDIX  F 


EXCURSIONS  ON  THE  STATISTICAL  ANALYSIS  OF  THE  RELATION  BETWEEN 
SE  INVESTMENTS  OVER  TIME  AND  CHANGES  IN  QUALITY  FOR 
RELATED  TIME  PERIODS 


Although  no  correlation  was  apparent  between  overall  SE  usage  and 
"quality,"  a  modestly  statistically  significant  relation  between  SE  in¬ 
vestments  over  time  and  changes  in  quality  for  related  time  periods  was 
observed.  The  following  text  and  tables  present  the  regression  analyses 
supporting  this  conclusion. 

Only  six  of  the  aircraft  in  the  sample  (F-14,  F— 1 5 ,  F-18,  F-5E, 

E-3,  and  P3-C)  had  data  available  for  both  quality  and  SE  usage  in  the 
first  few  operating  years;  thus  the  analysis  was  limited  to  just  those 
six  aircraft.  In  order  to  enlarge  the  data  set,  a  pooled  sample  of 
those  six  aircraft  over  two  time  periods  was  created.  A  simple  F-test 
suggested  that  the  data  sets  from  those  two  time  periods  were  from  the 
same  population,  so  the  data  could  be  pooled  to  create  a  set  of  12 
observations . 

Although  the  F-test  indicated  that,  taken  as  a  whole,  the  two  data 
sets  are  not  statistically  different,  individual  coefficients  may  differ 
based  on  a  T-test.  (The  F-test  and  7-test  sometimes  give  different 
results.)  To  examine  that  possibility,  the  three  basic  variables  ( Delta 
se,  se  -  first  year,  and  new )  were  entered  in  a  regression  equation  to¬ 
gether  with  a  binary  variable  (P)  equal  to  1  for  the  first  period  and  0 
for  the  second  period.  In  addition,  each  basic  variable  was  entered 
multiplied  by  the  period  1  binary  variable.  Table  F-1  shows  the  raw 
data  and  table  F-2  the  regression  results. 

The  variables  formed  as  a  product  using  the  period  1  binary  varia¬ 
ble  (such  as  p(5F1))  are  the  ones  relevant  for  testing  for  differences 
between  the  two  data  sets.  For  instance,  the  coefficient  on  p(new) 
estimates  the  difference  between  the  coefficient  on  new  in  the  period  1 
data  set  and  the  coefficient  on  new  in  the  period  2  data  set.  Simi¬ 
larly,  the  7  value  on  p(new)  is  the  relevant  one  for  testing  the  signif¬ 
icance  of  this  difference.  Table  F-3  summarizes  (to  two  digits)  the 
coefficients  and  7  values  relevant  for  analyzing  the  differences  between 
the  two  periods. 


1.  SE  investments  are  the  appropriate  SE  usage  values  for  the  given  time 
period  adjusted  for  SE  in  the  initial  year  and  aircraft  status  (i.e., 
wholly  new  aircraft  versus  model  change  aircraft). 


TABLE  F-1 


VARIABLES 


Aircraft 

Per iod 

Del t a  Q 

Delta  SE 

NEW 

SEj 

p 

P-3C 

1 

-0.26 

1.19 

0 

0.98 

i 

F-5E 

1 

-  -0.70 

0.93 

0 

0.61 

1 

E-3 

1 

0.70 

0.97 

1 

1 .06 

1 

F  —  1 8 

1 

0.40 

1.18 

1 

1 .52 

i 

F-1 5 

1 

0.53 

0.89 

1 

1 .04 

1 

F-1 4 

1 

-0.66 

0.96 

1 

1  .37 

1 

P-3C 

2 

-0.90 

1.02 

0 

0.98 

0 

F-5E 

2 

0.58 

1.10 

0 

0.61 

0 

E-3 

2 

-0.59 

0.92 

1 

1  .06 

0 

F- 18 

2 

0.69 

1.07 

1 

1.52 

0 

F— 1 5 

2 

0.41 

1.04 

1 

1  .04 

0 

F-1 4 

2 

-0.21 

0.89 

1 

1-37 

0 

Period 
Delta  Q 
Delta  SE 
NEW 

sjs1 

P 

PiDelta  SE) 
P( NEW ) 
P(SE}) 


time  period 

change  in  quality  for  time  period 

actual  SE  divided  by  "target"  SE  for  time  period 

1  for  new  aircraft;  0  for  model  change 

SE  usage  for  first  year 

dummy  variable  for  distinguishing  time  periods 
(1  =  time  period  1;  0  =  time  period  2) 

p  multiplied  by  Delta  SE 

p  multiplied  by  new 

p  multiplied  by  SE-, 


F-2 


TABLE  F-2 


ANALYSIS  OF  VARIANCE— DEPENDENT  VARIABLE  DELTA  Q 


Source 

DF 

Sum  of 
squares 

Mean 

square 

f  value  Prob  >  f 

Model 

7 

3.45 

0.49 

3.01  .15 

Error 

4 

0.65 

0.16 

Corr.  total 

11 

4.11 

Root  MSE 

0.40 

R-square 

0.84 

Dep  mean 

0.00 

Adj  R-sq 

0.56 

PARAMETER  ESTIMATES 


Var  iable 

DF 

Parameter 
est imate 

Standard 

error 

T  for  HO: 
parameter  =  0 

Prob  >  |r| 

Intercep t 

1 

-7.88 

2.75 

-2.87 

.05 

SE  i 

1 

-0.28 

0.83 

-0.34 

.75 

NEW 

1 

0.96 

0.55 

1 .76 

.  15 

Delta  SE 

1 

7.50 

2.48 

3.02 

.04 

P(5S1 ) 

1 

-4.13 

1.91 

-2.17 

.  10 

P(  NEW ) 

1 

2.21 

1.14 

1 .94 

.  12 

P(Delta  SE) 

1 

0.04 

3.84 

0.01 

.99 

P 

1 

2.92 

3.43 

0.85 

.44 

TABLE  F-3 

COEFFICIENTS  RELEVANT  FOR  TESTING 
FOR  DIFFERENCES  BETWEEN  SAMPLES 


Variable 

Coefficient 

T  value 

T  prob 

p(se1 ) 
p(new) 

-4.13 

-2.17 

.  10 

2.21 

1.94 

.  12 

P{Delta  SE) 

0.04 

0.01 

.99 

P 

2.92 

0.85 

.44 

F-3 


The  coefficient  on  p(sfi^)  of  -4.13  means  that  in  data  set  1,  the 
coefficient  on  se,  differs  from  the  corresponding  coefficient  in  data 
set  2  by  -4.13.  The  T  value  on  that  difference  (-2.17)  is  significant 
at  the  10-percent  level,  though  not  at  the  5-percent  level.  The  inter¬ 
pretation  of  the  coefficients  in  cable  F-3  as  showing  differences  be¬ 
tween  the  data  sets  can  be  confirmed  by  looking  ahead  to  tables  F-4  and 
F-5,  which  show  the  regressions  separately  for  the  two  data  sets.  Simi¬ 
larly,  the  coefficient  on  p(new)  suggests  that  the  coefficient  on  new  in 
period  1  is  2.21  greater  than  that  in  period  2.  From  the  T  value  (1.94) 
it  can  be  concluded  that  the  difference  is  almost  significant  at  the 
10-percent  level. 

The  tiny  coefficient  and  T  value  on  p{Delta  se )  indicate  that  the 
coefficients  on  Delta  se  are  virtually  identical  in  the  two  data  sets. 
Similarly,  the  insignificance  of  the  coefficient  on  p  suggests  that  the 
intercepts  in  the  two  data  sets  are  significantly  different. 

In  sum,  table  F-3  indicates  that  in  the  two  data  sets  the  coeffi¬ 
cients  of  Delta  se  are  very  similar.  The  coefficients  on  new  and  S£1 
are  quite  different.  The  estimated  differences  run  in  the  same  direc¬ 
tion  as  the  effects  in  period  1.  Hence,  the  differences  suggest  a 
diminished  effect  of  S£-|  and  new  during  the  second  period.  The  differ¬ 
ences  are,  perhaps,  even  more  apparent  in  the  separate  regressions  for 
the  two  periods  (tables  F-4  and  F-5). 


TABLE  F-4 

ANALYSIS  OF  VARIANCE— RESULTS  FOR  FIRST  PERIOD 

(Dependent  variable,  DIF42 ) 


Source 

DF 

Sum  of 
squares 

Mean 

square 

f  value  Prob  >  f 

Model 

3 

1.85 

0.62 

17, 

.37 

.05 

Error 

2 

0.07 

0.04 

Corr.  total 

5 

1.92 

Root  MSE 

0.  19 

R-square 

0, 

.96 

Dep  Mean 

0.00 

Adj  R-sq 

0, 

.91 

PARAMETER  ESTIMATES 


Var iabie 

DF 

Parameter 

estimate 

Standard 

error 

r  for  HO: 
parameter  =  0 

Prob  >  |  T 

Intercept 

1 

-4.96 

0.95 

-5.20 

.04 

se2-3 

1 

7.54 

1.37 

5.52 

.03 

NEW  J 

1 

3.17 

0.46 

6.83 

.02 

S£1 

1 

-4.42 

0.80 

-5.52 

.03 

F-4 


TABLE  P-5 


ANALYSIS  OF  VARIANCE  — RESULTS  FOR  SECOND  PERIOD 

(Dependent  variable,  DIF64 ) 


Source 

DF 

Sum  of 
squares 

Mean 

square 

f  value  Prob  >  F 

Model 

3 

1.60 

0.53 

1 .83 

.37 

Error 

2 

0.58 

0.29 

Corr.  total 

5 

2. 18 

Root  MSE 

0.54 

R-square 

0.96 

Dep  Mean 

0.00 

Adj  R-sq 

0.91 

PARAMETER  ESTIMATES 


Var iable 

DF 

Parameter 

estimate 

Standard 

error 

T  for  HO: 
parameter  =  0 

Prob  > 

Intercept 

1 

-7.88 

3.67 

-2.15 

.16 

S£4-5 

1 

7.50 

3.31 

2.26 

.15 

NEW 

1 

0.96 

0.73 

1.32 

.32 

5£’1 

1 

-0.28 

1.11 

-0.26 

.82 

Table  F-6  shows  the  basic  regression  applied  to  a  single  quality 
improvement  increment  from  year  two  to  year  six.  Thus,  it  provides 
information  similar  to  that  from  the  pooled  sample.  The  coefficient  on 
Delta  se  ( SE2_c) )  in  this  regression  is  the  same  as  that  displayed  in 
table  8  of  the  main  text. 


r 


F-5 


TABLE  F-6 


ANALYSIS  OF  VARIANCE— RESULTS  SPANNING  TWO  PERIODS 


(Dependent 

variable, 

DIF62) 

Sum  of 

Mean 

Source 

DF 

squares 

square 

F  value  Prob  >  F 

Model 

3 

3.33 

1.11 

2.56  .29 

Error 

2 

0.87 

0.43 

Corr.  total 

5 

4.20 

Root  MSE 

0.66 

R-square 

0.79 

Dep  Mean 

0.00 

Adj  R-sq 

0.48 

PARAMETER  ESTIMATES 


Variable 

DF 

Parameter 

estimate 

Standard 

error 

T  for  HO: 
parameter  =  0 

Prob  > 

Intercept 

1 

-9.47 

4.70 

-2.01 

.  18 

SE2-5 

1 

11.60 

5.53 

2.  10 

.  17 

NEW 

1 

3.63 

1.36 

2.67 

.  12 

SE  i 

1 

-4.30 

2.10 

-2.04 

.18 

I 


ft 


1 

f 


F-6 


